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ABSTRACT 


Human performance in spatial orientation tasks is mainly determined by 
spatial awareness and the skills to transition from the current spatial attitude into 
the desired spatial orientation and position. Erroneous spatial awareness may 
lead to degraded task performance, to the loss of equipment, to serious injuries, 
or fatal aviation mishaps. 

The use of UAVs is considered beneficial due to the reduction in risk to 
the human carrying out the “mission”. However, the remote execution of such a 
mission is extremely demanding for the operator. If extensive use of UAVs is to 
become routine, a number of concerns that may influence their effective use 
needs to be addressed. When we consider the human-in-the-loop (HITL), then 
vehicle control and the use of autonomy are important issues for the end user. 

Therefore this thesis will investigate the use of a virtual avatar in the flight 
simulator software (Weber Box) and conduct experimental proof of concept 
(conduct of experiments and analysis, evaluation and validation of the data of the 
concept using actual flight simulation software). Results of a study (conducted by 
Weber, 2006) indicated that the proposed design (Weber Box) seemed to 
strongly support spatial awareness in 3D orientation tasks. Time to assess a 
spatial situation decreases significantly, whereas accuracy of this spatial 
judgment at least maintains its level. 

This study investigated human orientation performance in relation to 
display designs that support mental models of the user’s spatial situation under 
varying workload conditions. The main goal is to support the pilot/operator with 
intuitive, 3D-based information which improves theirs spatial awareness and 
supports their mental model of spatial position, he/she is operating under, even 
with varying workload conditions. As a follow-up study has to be identified, 
determining whether varying workload affects performance between the two 
display designs, and if there is a significant difference to a set of properties which 
are essential for linking virtual avatars and spatial awareness. 
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I. INTRODUCTION 


A. OVERVIEW 

Most aircraft disasters are the result of inappropriate action or actions 
carried out by the pilots involved; a process known as "pilot error." Aircraft 
disasters are usually the outcome of a chain of events where many of the factors 
involved are insignificant in their own, but when combined with each other or with 
other factors they can result in a situation whereby the pilots involved become 
confused as to what is happening (a loss of situational awareness) and they 
begin to react in a way that the situations end up with catastrophic results 
(Paterson, 2000). 

A significant causal factor as to why pilots become confused and begin to 
mishandle the situation in which they find themselves, is among many others 
poor cockpit design. Simple visual cues increase human awareness and 
perception and decrease reaction times. Humans are visual beings requiring 
visual cues to warn them of impending danger, especially in combat aviation. The 
simplest cues are those that allow individuals to immerse themselves in the 
situations to which they involve and must respond. Two-dimensional (2-D) 
display plan-views of aircraft attitude have real limits on what types of information 
and how much information they can present to the viewer without becoming 
disorienting or confusing. Situational Awareness requires a transition from 2-D to 
3-D display perspectives. 

The large number of aircraft accidents due to spatial disorientation lead to 
the opinion that these accidents may be attributed to improper instrument use 
and interpretation (Clay, 1993). The need to fly without visual references was an 
additional incentive to improve flight instruments. Technological breakthroughs 
enabled the development of multifunctional instruments and the so called glass 
cockpit (Mejdal and McCauley and Beringer, 2001). 
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The basic motivation for this thesis is the high impact of spatial 
disorientation (SD) on aviation mishaps. SD has become a major issue in 
modern fixed wing and rotor wing aviation mishaps prevention. Mishaps due to 
spatial disorientation claim nearly three times more lives than non-SD mishaps 
(Matthews, Previc, and Bunting, 2003). 

Studies have shown that maintaining spatial awareness and preventing 
spatial disorientation is important for operating Unmanned Aerial Vehicles 
(Matsangas, 2004). Relationships between workload and operating UAV 
performance can be measured. Results of a study conducted by Weber, (2006) 
indicated that the proposed design (Weber Box) seemed to sustain remarkably 
well the spatial awareness in 3-D orientation tasks. Time to assess a spatial 
situation decreases significantly, with no loss in accuracy of this spatial judgment 
at least maintains its level. Judgment errors were minimized and the extreme 
errors were almost eliminated. 

The purpose of the current study was to assess both subjective workload 
and associated operating UAV control performance under varying workload 
conditions for both experienced and novice operators/pilots. Specifically, the 
focus was to assess the relationship between subjective workload and various 
objective measures of simulated operation/performance both with and without the 
introduction of a secondary task involving operating more than one UAV 
simultaneously. Two experiments were conducted to investigate how the 
proposed design affects operators’ orientation performance and to test the new 
effectiveness of the design concept under varying workload conditions. 

B. THESIS ORGANIZATION 

Chapter II reviews literature covering the major concepts, issues and 
systems underlying Situation Awareness, spatial awareness and spatial 
disorientation. The methods used are presented in Chapter IV. Chapter V covers 
the analytical strategy and presents the statistical results. Finally, conclusions 
and recommendations for future research are offered in Chapter VI and VII 
respectively. 
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II. LITERATURE REVIEW 


A. SITUATION AWARENESS 

Situation Awareness (SA) is a term used originally in the aircraft 
community and achieving it is perhaps the one of the most difficult aspects of an 
operator’s work. It has been defined by Endsley (1988) as: “...the perception of 
the elements within a volume of time and space, the comprehension of their 
meaning, and the projection of their status in the near future.” It is therefore clear 
that SA is necessary in order for people to perform tasks effectively (Endsley, 
2000). Dominguez (1994) gives a similar definition, but places more emphasis on 
the impact of awareness on cue extraction and directed perception (i.e. its 
contribution to attention). For Dominguez (1994) then, SA constitutes a 
continuous extraction of environmental information, integration of this knowledge 
to form a coherent mental picture, and the use of that mental picture in directing 
further perception and anticipating future events. 

Situation Awareness is most frequently defined in operational terms. 
Situation Awareness is knowing what is going on around you. We have been 
concerned mainly with people who need Situation Awareness for specific 
reasons, rather than individuals who has been largely outside the scope of 
human factors design efforts. For a given operator, therefore, SA is defined in 
terms of the goals and decision tasks for that job. The pilot does not need to 
know everything, but he/she does need to know at least the information to 
achieve a safe flight whether he/she is flying an aircraft or he/she operates a 
UAV as the related goal respectively. Although the "elements" of Situation 
Awareness vary widely between domains, the nature of SA and the mechanisms 
used for achieving SA can be described generically. 

Shown in Figure 1 this definition helps to establish what "knowing what is 
going on" means. 
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• Stress & Workload 

• Complexity 
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The enhancement of operator Situation Awareness has become a major 
design goal for those developing operator interfaces, automation concepts and 
training programs in a wide variety of fields. 

Today's systems are designed and therefore capable of producing a huge 
amount of data, both on the status of their own components, and on the status of 
the external environment. Due to achievements in various types of data link and 
communications, systems can also provide data on almost anything anywhere in 
the world. The problem with today's systems is not a lack of information, but 
finding what is needed when it is needed fulfilling the principals “least to know” 
and “need to know”. 

Unfortunately, there is a gap between the amount of data being produced 

and disseminated and the people's ability to find the information that is needed 
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and process it with the rest of the arriving information and the actual information 
that is required for their decisions. This information must be integrated and 
interpreted correctly as well. Issues of automation and "intelligent systems" have 
frequently only exacerbated the problem, rather than aided it (Endsley and Kiris, 
1995; Sarter and Woods, 1995). In addition to designing systems that provide the 
operator with the needed information and capabilities, we must also insure that it 
is provided in a way that is useable cognitively as well as physically. We want to 
know how well the system design supports the operator's ability to get the 
needed information under dynamic operational constraints (i.e. How well does it 
bridge the information gap?). This design objective and measure of merit has 
been termed “Situation Awareness.” As an expansion, the previous general 
definition of SA (Endsley, 1988) has been found to be applicable across a wide 
variety of domains as discussed below: 

1. Level 1 SA - Perception 

In this first degree of SA, the perception of the elements in the 
environment, humans achieve SA by perceiving the status, characteristics, and 
dynamics of all relevant elements in the environment. For instance, pilots have to 
perceive important elements such as status of the own aircraft, other air traffic, 
terrain, flight parameters and warnings. 

First, perception of cues (Level 1 SA) is fundamental. Without basic 
perception of important information, the odds of forming an incorrect picture of 
the situation increase dramatically. Jones and Endsley (1996) found that 76% of 
SA errors in pilots could be traced to problems in perception of needed 
information (due to either failures or shortcomings in the system or problems with 
cognitive processes). In military aviation, the number of tasks and the rate of 
change of the environment is very high because of the dynamic flight maneuvers, 
ground- and airborne-enemy activities, sophisticated weapon systems etc. 

2. Level 2 SA - Comprehension 

The second level of SA is comprehension of the current situation and is 
based on synthesis Level 1 elements. The key for Level 2 SA is that the human 
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must integrate Level 1 information to form a complete mental representation of 
the environment. Situation Awareness as a construct goes beyond mere 
perception however. It also encompasses how people combine, interpret, store, 
and retain information. Thus, it includes more than perceiving or attending to 
information, but also the integration of multiple pieces of information and a 
determination of their relevance to the person's goals (Level 2 SA). 

This is analogous to having a high level of reading comprehension as 
compared to just reading words. Twenty percent of SA errors were found to 
involve problems with Level 2 SA (Jones and Endsley, 1996). Flach (1995) points 
out that "the construct of Situation Awareness demands that the problem of 
meaning be tackled head-on. Meaning must be considered both in the sense of 
subjective interpretation (awareness) and in the sense of objective significance or 
importance (situation)." A person with Level 2 SA has been able to derive 
operationally relevant meaning and significance from the Level 1 data perceived. 

3. Level 3 SA - Projection 

The third level of SA, the projection of future status, is the highest level of 
SA. It enables a person to predict, at least in the very near future, the evolving 
situation. This is achieved through excellent knowledge of the functioning and 
dynamics of the system and a comprehension of the situation (both Level 1 and 
Level 2 SA). Pilots have to constantly project their current situation into the future 
and will base their decisions on this prediction. 

At the highest level of SA, the ability to forecast future situation events and 
dynamics (Level 3 SA) marks operators who have the highest level of 
understanding of the situation. This ability to project from current events and 
dynamics to anticipate future events (and their implications) allows for timely 
decision making. 

4. SA as a Multidimensional Construct 

The fact that SA is not a single ability creates serious problems for human 
factors researchers because it means they have no single measure of the 
phenomenon in which they are most interested. Indeed when Dennehy and 
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Deighton (1997) reviewed the literature from the period of 1979 to 1992, they 
identified no less than 28 different variables which had been used as measures 
of SA. Considering that these variables, Dennehy and Deighton set out to find 
the main SA subscales or "meta-categories," as follows: 

• SA Meta-Category 1 - Pilot Knowledge: The first principal 
component was a composite measure of knowledge, per se. 

• SA Meta-Category 2 - Anticipation and Understanding of Future 
Events: The second principal component was a composite measure 
of an ability cluster for anticipating and understanding future events. 

• SA Meta-Category 3 - Capacity to Manage Stress, Effort, and 
Commitment: The third principal component was a composite 
measure of an ability cluster for managing stress and maintaining 
effort and commitment. 

• SA Meta-Category 4 - Capacity to Perceive, Attend, Assess, and 
Assimilate Information: The fourth principal component was a 
composite measure of an ability cluster for obtaining and 
processing information from the world itself, both directly and from 
the cockpit dials and gauges available. 

• SA Meta-Category 5 - Overall Awareness: The fifth principal 
component was a composite measure of general awareness, 
similar to the "g-factor" popular among some intelligence theorists. 

In a further attempt to understand this clustering, Dennehy and Deighton 
adopted the Endler-Mischel "Interactionist Theory" (Endler, 1973; Mischel, 1973). 
This theory holds that it is impossible "to understand the role of an operator in 
isolation from the context within which she/he operates" (Dennehy and 
Deighton). The result is a "P-E Fit" model, one which tries to "fit" the person into 
his/her environment. Dennehy and Deighton characterized SA as the 
"operational space" provided by the two interacting domains in Figure 2. 
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Figure 2. The Interactionist ("P-E Fit") Model of Situational Awareness (From 

Dennehy and Deighton 1997) 

However, Endsley (1995, 1999) points out that it is simultaneously 
possible to divide up a pilot's declarative knowledge base (Dennehy and 
Deighton's first meta-category) according to the type of knowledge involved, as 
follows: 

• Geographical SA: This knowledge domain stores facts about the 
physical world, for example, the heights of mountains, the nature of 
the terrain, the locations of alternative airfields, the direction of the 
prevailing wind, etc. 

• Spatial/Temporal SA: This knowledge domain stores facts about 
the four-dimensional physical world (the three spatial dimensions, 
plus time), for example, aircraft position and speed, time into 
mission, etc. This is the most important for the current study. 

• System SA: This knowledge domain stores facts about system 
states, for example, engine, control, or instrument malfunctions, 
fuel status, etc. 

• Environmental SA: This knowledge domain stores facts about 
weather and visibility, together with such things as restricted (keep 
out) or prescribed (keep in) airspace. 
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• Tactical SA: This knowledge domain stores facts about the identity 
and capabilities of all other units in the vicinity (and, if military, that 
will include their combat intention as well). 

Endsley brought all these concepts together into what he presents as an 

all-embracing theoretical model of SA (Figure 3). 
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Figure 3. Model of Situational Awareness (From Endsley 1995, 1999) 

Another approach is the so-called situated cognition model by Miller and 
Shattuck (2004), which describes SA model as a process, rather than certain 
fixed states as in Ensley’s model. It is based on the one hand, on Pew’s model of 
ideal SA, achievable SA and actual SA (Pew, 2000), which describes the 
problem of measuring SA. On the other hand, it includes the lens-model of 
Tucker and Hammond, of information selection, and filtering (Tucker, 1964). 
Miller and Shattuck’s model focuses on processes rather than states and 
includes both human and machine elements of a system. It is oriented toward 
assessing human-system performance, tracks the evolution of activities and 
cognition, and links it to cognitive decision processes. From this perspective it 

describes the process of building SA starting at the ground truth, which is 
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selectively perceived by different layers of sensors (lenses). Every lens filters and 
transforms the received information. Then, the filtered information is processed 
by methods similar to Endsley’s 3-Levels-of-SA model. In Miller and Shattuck’s 
model, these levels are not independent levels of SA (Figure 4). They are, rather, 
specific cognitive processes which map additional aspects of the task and the 
environment like goals, expectations, personality etc. to the originally perceived 
information (Miller and Shattuck, 2004). 


Dynamic Model of Situated Cognition 



2004) 


SA requires an operator to “quickly detect, integrate and interpret data 
gathered from the environment.” In real-world conditions, Situational Awareness 
is hampered by two factors: data spread throughout the visual field and data that 
is noisy (Green, Odom and Yates, 1995). Understanding how data flow through 
the system, where the data may have been blocked, and how lenses may be 
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skewed is vital to knowing how to redesign systems and develop appropriate 
training (Miller and Shattuck, 2004). One of the greatest challenges facing the 
operators and technology providers is to match our increasing capability to 
gather, process, and display Situational Awareness information with a 
corresponding increase in our ability to use that information (Marsh, 2000). 
Increased availability of information about the situation may also be 
counterproductive with respect to time critical decision making. When decision 
makers expect information to be sparse, they tend to rely on judgment and 
experience to fill the voids. When information is massive and confusing, decision 
makers may often delay taking action until additional information can be gathered 
to fill voids or resolve ambiguities. They may reorient their decision making 
process from judgment based on experience to reliance on detailed analysis 
based on hard data. The result can be “analysis paralysis,” leading to a delay of 
the decision until it is too late (Marsh, 2000). Designers can help avoid these 
pitfalls by using properly designed technology coupled with well-trained 
operators, which will result in optimal human-system performance (Miller and 
Shattuck, 2004). 

5. How Do We Get SA? 

SA is the product of all various sources of information, as shown in Figure 
5. Cues may be received through visual, aural, tactile, and olfactory or taste 
receptors. As we move towards the instantiation of remote operators in many 
domains (e.g., unmanned air vehicles, remote maintenance, etc.), a major 
challenge will be providing sufficient information through a remote interface to 
compensate for the cues once perceived directly (Endsley, 1995). 
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Figure 5. Sources of SA Information (From Endsley 1995c, 1997) 

It is important to note that there is a tendency to focus on the information 
provided through the system and its operator interface which is not the only 
source of SA. In many domains, operators may be able to directly view and hear 
information from the environment itself, although in some cases they may not. It 
is important therefore that analyses of the SA provided by system designs also 
take into account that information that operators derive from other means. That 
is, what is the value added (or subtracted) by a given system taking into account 
what information one already gets via other means and what it may cost in terms 
of interference with that information (Endsley 2000). Of all the data the system 
possesses, some portion is displayed to the operator via its user interface. Of this 
information, the operator perceives and interprets some portion, resulting in SA. 
The role of others in the process of developing SA has also received attention. 
Verbal and non-verbal communication with others (including radio 
communication, hand signals and "wing-tipping" by other pilots) has historically 
been found to be an important source of SA information (Endsley 2000). 
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B. SPATIAL AWARENESS 

1. Spatial Awareness 

To gain a better understanding of the term spatial awareness, we define 
its meaning and other terms used in a similar context. Spatial Awareness is best 
understood as an important component of Situational Awareness. In a flight 
environment, developing Situation Awareness involves an assessment of 
numerous factors both internal and external. In order to identify the presence, 
magnitude, and possible intentions of a perceived threat, the operator must 
assess various aspects of aircraft behavior such as range, heading, altitude, 
speed, attitude (climbing or descending) and location on earth. In order to 
evaluate alternative responses to the potential threat the operator must take into 
account the status of defensive assets, considering factors that may limit 
available options, such as equipment malfunction or damage, fuel availability, 
atmospheric conditions, and terrain. All of these evaluations are made within a 
constantly changing environment and under highly stressful situations in which 
timing is critical and errors may be catastrophic. 

Spatial awareness refers to an operator’s comprehension of the 3D 
geometry of the environment in which he/she is operating. Three-dimensional 
information contained in the environment includes the absolute distance of 
objects (distance from an observer to an object), the relative distance of objects 
(distance between one object and another object or the distance between 
different parts of a single object), and the true 3D shape of objects (Wickens, 
Todd and Seidler, 1989b). In a flight environment, this information is available to 
an operator directly from his forward field of view and other senses and from 
visual displays (Endsley, 1988). In some instances, for example at night or when 
objects are out of viewing range, the operator may have to rely solely on visual 
displays for a spatial representation of the environment (Andre, Wickens, 
Moorman and Boschelli, 1991). Visual displays are therefore critical to operators’ 
spatial awareness. 
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Currently, most operators rely on plan-view displays to develop mental 
models of the space in which they are operating. A major limitation of this type of 
display is that it can only represent information from two dimensions of space; 
the vertical dimension is typically encoded in a textual format. To obtain 
information about the vertical dimension of a track in the environment, operators 
are required to “hook” the track and press a button to obtain textual readouts of 
altitude. To determine aircraft attitude, operators must monitor altitude readouts 
over time and observe changes. Operators are therefore forced to integrate 
textual with spatial information and mentally reconstruct the 3D nature of the 
visual scene. This process requires valuable cognitive resources and decision¬ 
making time (Haskell and Wickens, 1993). These limitations of plan-view displays 
may be overcome by 3D display technology. Three-dimensional displays can 
depict all three dimensions of space in a completely spatial format thereby 
eliminating the requirement to integrate textual with spatial information. All of this 
information is contained within a single display which reduces the need for 
mental integration of information from multiple sources (Woods, 1984). Three- 
dimensional displays also provide a more natural or ecological representation of 
the “real world” (Wickens et al., 1989b). Three-dimensional computer graphics 
systems include perspective displays, stereoscopic displays, rotating displays, 
head-motion tracking displays, holographic displays, and multi-plane displays. 

Ellis, McGreevy and Hitchcock (1984, 1987) examined how pilots’ 
avoidance maneuvers varied as a function of display type: plan-view versus 
perspective. Results from 10 airline pilots showed that the perspective display 
produced improved avoidance maneuvering; pilots took less time to identify 
collision hazards and recommend a maneuver, fewer errors were made in 
selecting a maneuver. Pilots were twice as more likely to select a vertical 
maneuver with the perspective display probably due to the more natural 
presentation of vertical separation. Bemis, Leeds and Winer (1988) compared a 
plan-view with a perspective display for the task of detecting an airborne threat 
and selecting the closest friendly aircraft to intercept the threat. The results from 
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21 naval operational personnel showed that fewer errors in detecting threats 
were made in the perspective display condition. The subjects were also more 
accurate and quicker at intercepting aircraft using the perspective display. Survey 
results showed that 19 of the 21 subjects preferred the perspective display. 

2. Models of Spatial Awareness 

According to the common definition of SA, in a three-dimensional (3D) 
orientation task, a person who has good SA has fast access to an accurate 
mental representation of the altering environment and may be able to predict his 
spatial situation in the near future. For example, an aviator with excellent SA may 
not be consciously thinking about the fact that there is an aircraft nearby. He is 
able to operate the aircraft proactively and correctly according to this situation. 
The pilot will do so quickly and precisely based on the ability to rapidly access 
the information from memory (Wickens, 2002). Thus, instrument design that 
supports SA will facilitate the appropriate reaction in uncertain situations. 

Spatial awareness can be seen as a subset of Situation Awareness or 
rather a specific application of it and it has many, often vague definitions. For this 
study, we consider spatial awareness as an extension of spatial orientation, 
where spatial orientation is understood as the ability to recognize an object’s 
orientation or position in three-dimensional space. It describes spatial awareness 
as the knowledge of their position relative to the desired flight route and the 
ground and its spatial attitude. In addition it includes the knowledge about the 
current flight maneuver; its own spatial behavior. 

Wickens (2002) pointed out that a pilot’s spatial awareness is determined 
by six crucial variables: pitch, roll and yaw (slip) of the aircraft, altitude, deviation 
from a flight path, and position along a flight path. More factors directly and 
indirectly, need to be continuously monitored to build a complete spatial model. 
Two of the most obvious are airspeed and stall-indicator. In addition, knowledge 
is needed about other air traffic, ground and weather. All variables are cross- 
linked, which means that one or more variables influence other variables in the 
future. For instance the combination pitch and airspeed may lead to later stalling 
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of the aircraft. Skilled pilots represent this linkage in a mental model and 
implement time constrains, latencies, and additional information within this 
model. 

3. Measuring Situational and Spatial Awareness 

An individual’s SA is a relatively difficult construct to measure (Hancock 
and Desmond, 2001). Another issue is to determine how much SA is sufficient for 
the task. Endsley states that SA and performance measures are only linked 
probabilistically, and that there are no set thresholds of SA to guarantee a given 
level of performance (Endsley, Sollenberger, and Stein, 2000). Endsley et al. 
therefore propose relative levels of SA. In terms of the design issue, this would 
mean an increase in SA (after the addition of new layout) relative to the level 
measured prior to this introduction. Measurement should also enable comparison 
to the “ideal” SA: that is, perfect knowledge on all relevant aspects of a situation, 
with no gaps or holes in it. As we discussed earlier, Pew states that actual SA 
tends to be a subset of the achievable SA, which is again a subset of the ideal 
SA of any situation (Pew, 2000). 

An individual’s level of SA will be partially determined by the quantity and 
quality of the information available and by the individual’s ability to utilize the 
important information sources and fill in for missing, imprecise, or incomplete 
information. Possible reasons for lowered SA are environmental factors such as 
lack of anticipation of vital factors (due to lack of awareness of the importance of 
these factors, or attention being focused on other things) (Entin, 1998). Entin 
refers to two types of SA measures: high-level and detailed SA. High-level SA 
measures facts according to subjects’ responses to general questions about the 
current situation. The detailed SA is measured with questions about elements of 
the situation based on a broad-ranging assessment of operator SA requirements) 
(Gawron, 2000). Detailed SA reduces when information changes rapidly. In 
aviation tasks, operators may not have time to integrate all the incoming 
information into a coherent picture. Basic types of SA measures derive from 
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Entin’s study of detailed and high level SA. One is based on the responses of 
subjects’ to questionnaires that included questions designed to capture specific 
elements of the situation. 

However, there are several other means by which SA can be measured, 
according to Pew (2000), which fall into the following categories. Direct systems 
performance measures are only applicable in very limited situations. Disruptions 
in order to disorient the operator can be introduced (Pew and Mavor, 1998). The 
operator’s recovery time and the success of recovery tactics taken can be 
measured. Performance measures are often not sufficient to diagnose human- 
system relationships in detail. In many cases we have to measure subjective 
facts as mental models or mental workload as well (Endsley et al., 2003). 

Subjective measures include self-assessments, expert judgments, peer 
ratings, and supervisor or instructor ratings. A different way to measure the 
gradient of change in SA is to infer it from objective performance measurements. 
In our opinion objective performance measurement should be conducted 
wherever possible. Subjective methods should be conducted to explain data of 
objective measurements. 

Process measures are considered as very effective in measuring SA. 
These may include eye-tracing, head tracking, information acquisition, flight 
parameter analysis and so on (Endsley et al., 2003). 

C. SPATIAL ORIENTATION - DISORIENTATION 

Spatial disorientation (SD) is the most common cause of human-related 
aircraft accidents. SD requires the knowledge of both the physiology and 
psychology of the human in flight and, to a lesser extent but still important, an 
understanding of the physics of an aircraft in motion. When reading about an 
accident involving SD, terms like visual illusions, vestibular misperceptions, task 
saturation, weather, motion, and aircraft experience are commonly found. In the 
past, researchers aimed much of the countermeasures research at improving 
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their understanding of these misperceptions. However, in order to better 
understand the overall situation, researchers must start with the SD definition 
and arrange each condition into distinct categories (Figure 6). 

The most widely used and accepted general definition is "A state 
characterized by an erroneous sense of one's position and motion relative to the 
plane of the earth's surface" (Freeman, et al, 1989-91). Prior to this accepted 
definition, researchers could not be certain that a particular incident qualified as 
SD or as another phenomenon altogether—often masking the real magnitude of 
the issue. To better support the use of this definition by researchers, pilots, 
physicians, and physiologists, an operational definition recently emerged—"An 
erroneous sense of the magnitude or direction of any of the aircraft control and 
performance flight parameters." (Gillingham, 1992). 



Figure 6. Inner ear with semicircular canals shown likening them to the roll, 
pitch and yaw axis of an aircraft (From www.atlasaviation.com ) 

Because of the different ways SD can occur, it is easier to study SD by 
separating it into three distinct categories: Type I—unrecognized, Type II— 
recognized, and Type III—incapacitating, where each type impacts the pilot in a 
different way (Heinle and Ercoline). 

The first group, Type I—unrecognized SD, explains the phenomenon as a 
state where the pilot is unaware of the flight parameters described in the 
operational definition. This is the most common type of SD due to many psycho- 
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physiological variables (e.g., task saturation, canalized attention, fatigue, etc.). 
Many operators state that this as a simple failure of the pilot to maintain an 
appropriate instrument crosscheck. An example of Type I SD is the post-roll or 
Gillingham Illusion. 

The second group of SD sensations encompasses those incidents that 
produce a recognized phenomenon known as a sensory mismatch or at least the 
awareness that something has gone wrong. This is labeled as Type II— 
recognized SD. An explanation of the classic Graveyard Spin Illusion 
demonstrates Type II SD (see Figure 7). 



Figure 7. Graveyard Spin Illusion (From www.atlasaviation.com ) 

In this example, the pilot enters a spin, becomes stabilized in yaw, and 
realizes a need to move the controls opposite to the direction of rotation. Once 
the pilot applies the opposite controls, the aerodynamic result is a decrease in 
the aircraft's angular rotational yaw followed by a false sensation of the aircraft 
beginning to spin in the opposite direction. When this occurs and if the pilot looks 
at the aircraft turn needle or compass card, the pilot experiences a sensory 
conflict. The turn needle will indicate a turn in one direction, while the inner ear 
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sensation will generate a feeling that the aircraft is turning in the opposite 
direction. When this occurs, the pilot often suspects an instrument malfunction 
and does not recognize the situation as SD. 

The third and last type of SD is the least common. Researchers call it 
Type III—incapacitating SD. Few studies of this type of SD exist, but researchers 
know that it does occur, through experience and pilot reports. An example of 
Type III SD is called the Giant Hand Illusion. The SD phenomenon has been 
intertwined with aviation since the beginning of manned flight, and only a 
concerted and coordinated research effort will make a difference in reducing SD 
mishaps. This effort begins with an understanding of the definition of SD, along 
with its three distinct types (Heinle and Ercoline). 

D. HUMAN PERFORMANCE AND COGNITION 

1. Human Perception and Cognition 

The term cognition is used in several different loosely related ways. In 
psychology it is used to refer to the mental processes of an individual, with 
particular relation to a view that argues that the mind has internal mental states 
(such as beliefs, desires and intentions) and can be understood in terms of 
information processing, especially when a lot of abstraction is involved, or 
processes such as knowledge, expertise or learning, for example, are at work. 
The term cognition also is used in a wider sense to mean the act of knowing or 
knowledge. It was derived from psychological science which attempted to explain 
human behavior and reasoning (Matthews, Davies, Westerman, and Stammers, 
2000). Unlike behaviorism, cognitive theory focuses on what is going on inside 
the person’s mind. Cognitive learning is not just a change in behavior; it is a 
change in the way a person thinks, understands, or feels. 

Perception is defined as a direct consequence of selective attention, which 
involves the extraction of meaning from an information set processed by the 
human senses (Wickens et al., 2004). In this study we focus on higher level 
perceptions which answer questions of “What am I in?”, “Where I am?”, “What is 
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my attitude?”, “Where am I going?” and so on (Warren and Wertheim, 1990). 
Hence, we start with thinking at a level above human sensory systems and 
physical stimulus. 

Endsley (1988; 1990; 1995) proposed a framework model based on 
information processing theory (Wickens, 1992). The cognitive mechanisms that 
are important for the development of SA are shown in Figure 8. 
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Figure 8. Mechanisms and Processes involved in SA (From Endsley 1988; 

1990; 1995) 

Wickens (1984) hypothesized that human attention capacity should be 
conceived as multiple resource pools, with dual-task interference being greatest 
when tasks compete for similar processing resources and least (or non-existent) 
when tasks draw from different resource pools. 

2. Human Performance Models 

Human performance models can be seen in three major tracks: Manual 

Control Models of human control in closed-loop systems, Task Network Models 

are concerned with fundamentally predicting the probability of success and 

performance time, and Cognitive Architectures Models are typically based on 

21 










theories of human performance capacities and limitations (Pew, 2000). The first 
two types of models arose from an engineering point of view and are based on 
control theory. Control theory describes any kind of system that has a feedback- 
loop. 

Cognition and human performance are linked in cognitive architecture 
models. In order to accomplish this we have to address more factors and 
connections. Matthews et al. (2000) call these major factors the “energetics of 
cognition” and associated with it capacity, resources, and attention. Capacity is 
understood as the physiological and physical limits of human performance. 
Resources refer to a limitation of attention. We might not have enough attention 
to follow many parallel things that are happening simultaneously. Resource 
theory is widely as a base theory to study attention. Attention has selective and 
intensive characteristics. The selective aspect refers to a choice of reaction to a 
certain stimuli rather than to others (Matthews et al., 2000). Divided attention is 
the ability to react to two things or more at the same time or to accomplish two or 
more tasks or mental activities (Wickens et al., 2004). 

We know that humans are not able to focus on a single object for more 
than some seconds. People have difficulty maintaining attention over a longer 
period. This is supported by various studies of the 1940’s and 50’s which did 
research on performance of radar and sonar operators and their detection time 
and error rate (Matthews et al., 2000). To explain sustained attention, four major 
models were developed over time: the filter theory, the expectancy theory, the 
arousal theory and the resource theory as described by Matthews et al. (2000). 

The filter theory predicts that vigilance tasks in which signals are present 
only for a short period will contribute to a faster decrease of attention than tasks 
with signals that are present throughout a longer period. 

According to the expectancy theory observers keep track of information to 
extrapolate the occurrence of future information. Over time, the observer’s 
internal estimate of the next information probability changes. 
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The basis of the arousal theory is that sustained attention leads to a lower 
level of arousal of the central nervous system (Matthews et al., 2000). It was 
shown by some experiments that cortical arousal level decays almost 
independently from the character of the task. 

Matthews et al. (2000) state that resource theory describes experiments 
that show that task workload plays a major role in vigilance decline. This theory 
suggests that performance in demanding tasks is more sensitive to resource 
limitations by fatigue than simpler tasks. This position is supported by a number 
of single and dual-task experiments (Wickens and Gosney, 2003). 

3. Working Memory and Attention 

Several factors influence the accuracy and completeness of Situation 
Awareness that individual operators derive from their environment. First, humans 
are limited by working memory and attention. The way in which attention is 
employed in a complex environment with multiple competing cues is essential in 
determining which aspects of the situation will be processed to form Situation 
Awareness. Information must be integrated with other information, compared to 
goal states and projected into the future - all heavily demanding on working 
memory (Endsley 2000). Several recent studies have confirmed the role of 
attention in Situation Awareness. Endsley and Smith (1996) found that fighter 
pilots' attention to targets on a tactical situation display was directly related to the 
importance of those targets in their tactical tasks. 

Attention to information is prioritized based on how important that 
information is perceived to be (Endsley 2000). Even experienced operators can 
make errors in this process, neglecting to attend to certain information over other 
information. Jones and Endsley (1996) found that the single most frequent causal 
factor associated with SA errors involved situations where all the needed 
information was present, but was not attended to by the operator (35% of total 
SA errors). This was most often associated with distraction due to other tasks. 
Correctly prioritizing information in a dynamic environment remains a challenging 
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aspect of SA. Good SA requires enough awareness of what is going on across a 
wide range of SA requirements (global SA) to be able to determine where to best 
focus one's attention for more detailed information (local SA). 

4. Mental Workload 

Consideration of the mental workload imposed by advanced technology is 
important to the military acquisition community and technology designers who 
must provide systems that do not overburden the user with operating tasks and 
information clutter. Resource or capacity models of mental workload postulate a 
limited quantity of resources available to perform a task, and in order to perform 
a task, one must use some or all of these resources. These models of workload 
address the difference between the amount of resources available within a 
person and the amount of resources demanded by the task situation (McCloy, 
Derrick and Wickens, 1983). Workload also refers to people’s experience of 
cognitive task performance as effortful or fatiguing, which may index task 
demands and attentional overload (Mulder, 1986). Measures of workload can be 
classified into four categories: primary task measures, secondary task measures, 
subjective measures and physiological measures. In this thesis, data were 
collected in the first three of these areas. 

5. Human Performance - Workload Measurement 

We defined human performance as a measurable outcome of a certain 
task conducted by humans. Human performance is complex in its representation 
and outcome. In many tasks it is difficult to describe the outcome parameters that 
are expected. For instance, an attack pilot has to accomplish a wide variety of 
sub-tasks in order to fulfill his main task: attack a certain object. He has to 
operate the aircraft, navigate relative to the ground and in space, to coordinate 
his actions with wingmen, search, and select the target, react to enemy fire, 
maintain radio communication with several sources, select the appropriate 
weapon, chose the right moment to fire the weapon etc. Furthermore, each of 
these tasks consists of various subtasks (Wickens, 2002). 
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The amount of information processing and decision-making required in 
task performance impacts the workload experienced by the performer. Hart and 
Staveland (1988) add to this definition that "Workload is not an inherent property, 
but rather it emerges from the interaction between the requirements of a task, the 
circumstances under which it is performed, and the skills, behaviors, and 
perceptions of the operator." 

Two major ways are used to measure human performance: objective 
measures and subjective measures. Objective measures are concerned with 
quantitative and qualitative outcomes of human performance. Therefore we 
measure physical variables like time and speed or we measure the quality of the 
outcome in terms of error rate or difference to the desired results. A subjective 
measurement assesses human performance through questionnaires or self- 
reports, which maybe considered as a weaker approach (Endsley et al., 2003). 
Objective performance based measurement, also called measures of 
effectiveness (MOEs) or measures of performance (MOPs) are considered as 
powerful tools for measuring the performance of a human-in-the-loop system and 
for identifying areas of inadequate Situation Awareness. The use of situations 
with testable responses can provide valuable insight into the user's Situation 
Awareness and how the user will act upon it (Endsley et al., 1998). 

Subjective ratings, are the scales that are the most widely used workload 
assessment tools. The advantages of subjective workload assessments are ease 
of implementation, low cost, and limited intrusion on task performance. They are 
also useful in evaluating the potential for task overload among competing 
interface designs. They can be applied during the design process with mock-ups, 
prototypes, and simulators, as well as assessing workload on existing systems. 
Two subjective rating tools frequently used in aviation and other safety-critical 
environments are the NASA Task Load Index (NASA-TLX) and the Subjective 
Workload Assessment Techniques (SWAT) which have subscales assessing 
loads for time, mental effort, and psychological stress. NASA-TLX allows users to 
perform subjective workload assessments on operator(s) working with various 
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human-machine systems. NASA-TLX is a multi-dimensional rating procedure that 
derives an overall workload score based on a weighted average of ratings on six 
subscales. These subscales: Mental Demands, Physical Demands, Temporal 
Demands, Own Performance, Effort and Frustration. It can be used to assess 
workload in various human-machine environments such as aircraft cockpits; 
command, control, and communication (C3) workstations; supervisory and 
process control environments; simulations and laboratory tests 
(www.nrl.navy.mil). 

Performance Data measures whether a portion or a user’s entire task is 
selected for objective and quantitative measurement. Traditionally, performance- 
based measurements have included speed and accuracy data. Speed can be 
reaction time either to perceive an event or to initiate a response. Accuracy can 
be defined as whether or not a signal is perceived and if an appropriate response 
is made. Physiological Measures use numbers to assess a cognitive workload. 
These measures, in general, fall into one of two classifications: Background 
Measures - those measures not linked to any specific ongoing tasks or to the 
timing of any user activity or response and Task-Related - those measures 
specific to a user activity, response, or an event. 

Questionnaires help to have a view into the cognitive model of the 
operator. By formulating the right questions the researcher is able to gain insight 
as to why the operator acted in one way or another. It helps to explain “hard” 
data of the objective measurement methods. One weakness is that 
questionnaires are subjective (Matthews et al., 2000), greatly depend on the way 
questions are asked and the experiment is organized to extract the desired 
information. In this study we use a set of performance-based measurement of 
spatial awareness and post-experiment questionnaires to evaluate subjective 
factors to better explain the objective measures (Prevot and Palmer, 2000). 
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III. PROPOSED DISPLAY DESIGN 


A. GENERAL BACKGROUND AND DESIGN OF WEBER BOX 

1. Background 

Navigation awareness, in particular, refers, as stated in previous chapters, 
to the pilot's dynamic representation or mental model of the aircraft's orientation 
within two dimensions, space and time (Endsley, 1988). Flight control is 
characterized by three spatial coordinates (x, y, and z) and three axes of rotation 
(pitch, roll, and yaw), providing 6 degrees of freedom in Euclidian space. 
Conventional display formats, however, use multiple two-dimensional (2-D) 
displays to provide orthogonal views of three-dimensional (3-D) flight information. 
The mental model is formed by the pilot's perception of his or her orientation in 
the outside world (e.g., attitude, heading, altitude, airspeed), which is based on 
information obtained through the aircraft displays, the forward field of view, 
and/or directly from the pilot's senses (Endsley, 1988). Several human factors 
arguments could be made for the implementation of 3-D technology displays: 
whether a 3-D view of the airspace provides a more "natural" representation than 
conventional plan view (2-D) displays, or if a single 3-D perspective display 
provides a more "compatible" view by reducing the need to integrate mentally 
across several 2-D displays (Woods, 1984). According to the initial design for 
Weber Box, pilots would benefit with information displayed to them that would 
help establish and monitor SA, enabling them, to correctly judge the proper 
attitude at any given time. Thus, Weber’s (2006) intention was to represent one 
display's information in the context of another (Aretz, 1991) in order to 
compensate for any wrong interpretation of the information provided. Besides the 
clear representation, a mapping of the real world is provided on an easy to learn 
set of instruments which creates less confusion and workload for the operator 
(pilot or UAV operator). Demanding less attention, the operator is able to reduce 
reaction time for tasks associated with distraction and interruptions. 
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2. Design Goals and Purpose 

The major design goal is to support the operator’s spatial awareness in an 
easily understandable and intuitive way, even when subjected to an increased 
workload. Traditional flight instrument design is based on historic design 
features, which are very abstract and non-intuitive (Figure 9). Conventional flight 
instrument displays require the pilot to scan instruments looking at or near each 
of a number of instruments in succession to obtain information. Over the years 
many different layouts have been experimented with; however there is now one 
accepted layout, which all modern airplanes adopt. This is known as the basic T. 



ARTIFICIAL HORIZON 


ALTIMETER 


AIRSPEED 


Figure 9. Traditional flight instrument design (Screenshot from X-Plane flight 

simulation) 

The common instrument design shown in Figure 9 demands extensive 
training to develop the necessary skills to interpret and use these displays 
(Headquarters Department of the Army, 1984). In extreme situations, fatal 
accidents have occurred because of misinterpretation of the cockpit instruments 
(Roscoe, 2002). 
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Because current aircraft display design inherits many poor design features 
from its historical roots, Weber (2006) decided to introduce a new way to 
represent major elements of flight dynamics. The purpose of the proposed design 
was to introduce an innovative way to display spatial information on flight 
instruments. The design concept therefore breaks with traditional flight 
instrument design. The display was not designed to substitute any of the 
traditional instruments. It rather supports the operator of an aircraft in extreme or 
ambiguous situations by intuitively representing an appropriate level of spatial 
awareness (Weber, 2006). 

3. Basic Design Concepts 

The overall display layout facilitates the implementation into HUDs or 
HMDs. Principles of abstract and simplified symbology and restrictive uses of 
colors are applied (Weber, 2006). 



Figure 10. Screenshot: overall display design (From Weber, 2006) 


The texture design of the aircraft was aimed to support definition of all 
possible spatial orientations. The top side has a green wire frame on black 
background; the under side has a black wire frame on green background (Figure 
12). Furthermore, the top side texture has only longitudinal lines, whereas the 
bottom side has squares. The application receives data from any flight simulator 
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via network communication using the UDP standard. The default UDP port was 
49001 while during the experiment it used the UDP port 49002. The UDP client 

TM 

demands the X-Plane data structure specifications. 
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IV. METHOD 


A. METHODOLOGY 

1. Experiment Goals and Purpose 

The goals of this study was to further implement, and evaluate the Weber 
Box proposed display design (Weber, 2006) that supports or enhances spatial 
awareness in 3D orientation tasks while having increased workload conditions. In 
order to accomplish this goal we have to assess the impact of the proposed 
display design on people’s spatial awareness in an environment that varies 
workload experimentally. 

2. Research Questions 

Due to the limited scope and assets of this study this evaluation sought to 
answer four main questions: (1) How will the display design, compared to a 
traditional display layout, influence the time to assess a static spatial orientation? 
(2) How will the same display design influence the time to assess a static spatial 
orientation while operating more than one UAV (3) What is the subjective 
impression of the design to the operator? (4) Will the impact and the subjective 
judgment be different for participants with a strong aviation background versus 
participants without an aviation background (pilots/non pilots)? 

The answer to the first two questions provides insight into how people are 
able to build their mental picture/model of the current spatial situation even when 
they have to operate under a varying workload environment. The time it takes 
them to mentally construct their spatial awareness enables us to come to 
conclusions about the effectiveness of the proposed design to support the mental 
process of generating spatial awareness. It allows us at least estimate the impact 
of the display on real world tasks and enables the participants to evaluate the 
design in the context of a goal-based task. 

The answer of the third question gives us insight about the impression of 
the participants beyond objective performance measurements. It enables us to 
bring the MOEs in context and may help to diagnose ambiguous outcomes. 
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The last question provides information about the relationship between 
personal background/training and the user acceptance of the proposed design. 
Perhaps a pilot with hundreds or thousands of hours of experience with 
traditional flight instruments will find it much easier to interpret and use the old 
instrumentation and harder to adapt to the new one operating under an 
increasing workload environment (multiples UAVs). The opposite might be the 
case for non-aviators. 

3. Constraints and Assumptions 

Since the experiment involves human participants we had to apply for 
approval by the Institutional Review Board (IRB) was required. The experiment 
was approved as proposed (Appendix A). 

The flight dynamics were provided by a commercial flight simulation 
program, X-Plane® by Laminar Research™ in version 7.13. X-Plane does not 
allow storing complete flight situations, including flight dynamics. We used the 
original cockpit layout generated by X-Plane, and a model of a Piper PA-46 
Malibu, which is a small single propeller turboprop aircraft. 

4. Technical Equipment 

The experiment took place at the MOVES Institute, Naval Postgraduate 
School in Monterey, California. The proposed display was implemented on a/two 
personal computer (Laptops) in addition with a/two 17-inch TFT monitor 
respectively, which was placed in front of the participant. All Laminar Research 
software uses OpenGL so a 3-D accelerator card that can run OpenGL was 
required to use the latest software. At minimum the configuration should be for 
Windows, Pentium 1ghz+, (or for Macintosh PowerMac 800+), with disk space 4 
gig, a 3-D card, VRAM at least 32 meg, and a monitor at a 1024x768+ resolution. 
A peripheral input device that looks similar to a UAV control device, a computer 
joystick (USB) was used that allowed the participants to control two UAVs similar 
to controlling an airplane in a flight simulator. The time required for the subject to 
assess spatial attitude for both sub-experiments was stopped by hand using a 
digital hand stopwatch type SPORTLINE®. 
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5. Data Collection Methodology 

Both sub-experiments stopped the time to assess the current spatial 
situation by hand with an estimated precision of ±0.1 seconds. The precision of 
the spatial judgments was measured by evaluating the drawings for pitch and roll 
out of the provided schema (Appendix F). The drawings were evaluated in 15° 
steps based on the error from the given values. Hence, the precision was ±15°. 
Thus, errors within 15° rounded to the next value. 

The subjective self-assessment provided a scale of five judgments in 
every questionnaire. Demographic data were collected by a pre-experimental 
questionnaire (Appendix E). Participants completed the NASA-TLX questionnaire 
immediately following each of the two sets. 

All participants were exposed to all instrument setups and all sub-experiments. 
The order of presentation of the two of displays was counterbalanced using a 
randomized subject assignment procedure. Selection was done as follows: 

• List participants names 

• Generate random numbers of enough digits that each exceeds the 
size of the sampling list by several digits. This makes duplication 
unlikely. 

• Assign the random numbers arbitrarily to individuals in the sampling 
frame list. 

• Sort the list of random numbers, carrying along the sampling frame 
list. 

• Now the first n values in the sorted sampling frame column are a 
simple random sample (SRS) of n values from the entire sampling 
frame. 

Odd-numbered participants started with the traditional instrument setup; 
even-numbered participants started with the new instrument setup. This ensured 
that learning effects were counter-balanced over the experiment. 

6. Data Analysis Methodology 

The data were recorded using spreadsheets. Every sub-experiment allows 
the participant six individually measured and recorded trials. The data were 
analyzed by commercial statistical software JMP® version 5.1.2, using various 
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methods e.g. linear regression analysis, residual analysis, ANOVA and paired t- 
tests. Two major questionnaires, subdivided into one pre-experimental 
demographic questionnaire, and two post-experimental subjective evaluation 
questionnaires where administered in this experiment (Appendix EandG). The 
questionnaires were given for several reasons: as to develop a secondary 
method to determine the effects of the dependant variables, and to enable a 
correlation between a participant’s subjective and quantitative data and to assess 
the relationship between subjective ratings of the workload and objective 
measures of performance between experienced and inexperienced (novice) 
operators. 

B. EXPERIMENT DESIGN 

1. Basic Experiment Design 

The experiment was designed to compare differences in time/speed and 
accuracy/precision while changing one variable (display design), while tested in 
different workload conditions (two UAVs). The dependent variables were the 
outcomes in terms of differences in time and accuracy. The independent variable 
was the instrument layout with two different designs, the traditional “basic T” and 
the new Weber Box. Participants and whether operating under varying workload 
conditions, meaning operating one or two UAVs simultaneously. 

The main idea was to have a set of two sub-experiments with static task in 
order to measure the time and accuracy of the participant’s assessment of their 
spatial situation. It also served the purpose of making the participants familiar 
with the use and interpretation of the instruments and displays. 

The participants were told that they were the operator of a combined rotor-/fixed- 
wing based Unmanned Aerial Vehicle (UAV). The UAV is on a test flight and 
enters a sector of bad weather. The data-link connection to the UAV was 
interrupted because of the weather conditions (simulated by switching off the 17” 
monitor). At given time the experimenter freezes the simulation software, 
providing equally difficult attitudes that have to be determined. 
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2. Experimental Procedure 

Every participant went through a series of experiment sets. First the 
participant was introduced to the experiment. Then the participant filled out the 
necessary paperwork, including consent form, privacy statement, and minimal 
risk information form (Appendix A). After the initial brief, the participant had the 
opportunity to become familiar with the flight simulator and control devices by a 
free self-paced flight training while assisted by the experimenter. 

After taking the pre-experimental training the participant started with task 
one and the given instrumentation setup (with or without the new instrument - 
Figure 11). Subsequent to this first sub-experiment, the participant starts the 
second sub-task, while this time he/she had to operate under an increased 
workload, i.e. controlling two UAVs. Each sub-experiment set was scheduled for 
20 minutes with an optional 10 minute break. 



EXPERIMENTER 


Figure 11. Instrumentation setup (task dependant respectively) 

When the participant finished the second experiment block, he/she was 
asked to fill out the post-experimental questionnaire. This finished the 
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experimental session for this subject. The detailed experiment is laid out in the 
experiment protocols (Appendix B). A pilot study was conducted with four 
volunteer participants to test procedures, apparatus, and tasks (having the same 
level of difficulty in defining the UAV’s attitude). 

3. Static Spatial Awareness Experiment 

The detailed procedure of this sub-experiment was documented in the 
experiment protocol. After the participant was led into the lab, he/she was shown 
all the equipment, and was instructed to sit in the operator place. The 
experimenter explained the use of all flight instruments and controls including the 
Weber Box, which was referred during the experiment as a working term to the 
proposed display. 

The participant was allowed to fly for about 5-10 minutes as a brief 
training to become familiar with controlling the UAV where explanations for all 
flight instruments and controls were still provided. After the training flight, the 
experimenter briefed the participant for the first task (Recognition of spatial 
orientation of one UAV, Appendix F). Then, the instrumentation monitor or the 
Weber Box respectively was switched off, depending on if he begins with or 
without using the Weber Box. 

The first trial was a test trial. The experimenter picked a set of pitch- and 
roll-angles from the set of possible angle pairs and transferred them into the flight 
simulation software. When the experimenter told him/her to start, while switching 
on/off respectively the monitors, he/she started to look at the instruments. When 
the participant said “Stop!” the experimenter measured the elapsed time and 
switched off the monitor. Then, the participant had to draw his/her opinion about 
pitch and roll into the provided schema on the evaluation sheet (Appendix F). 
The experiment started over with the next experiment test and a new set of pitch 
and roll angles until the participant finished six tasks (plus one test trial). 

4. Static Spatial Awareness Experiment with Increased Workload 

The detailed procedure of this sub-experiment was documented in the 

experiment protocol (Appendix B). As the participant was already led into the lab, 
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and was shown all equipment, while sitting in front of the 17” monitor he/she had 
only to perform the exact static spatial awareness experiment while this time 
he/she was asked to operate two UAV’s (increased workload). 

5. NASA-TLX over increased Workload 

The NASA-TLX (Task Load index) was scored for each participant as 
follows. First, the number of times each factor was circled in the top portion was 
counted to determine the weight for that factor. The total number of weights 
summed to 15. Second, a scoring template was made with the subscales divided 
into 20 equal portions, and that template was lined up behind each participant’s 
TLX sheet. The numbered portion in which the participant’s mark intersected the 
line determined the score for the subscales. In cases where the mark appeared 
exactly on the line dividing the sections, the higher number was given. In 
addition, in cases where the mark did not touch the line, a mark was 
extrapolated, although in most cases the majority of the mark was in the section 
assigned. In cases where the participant had written a checkmark or an “x,” the 
corner of the check or intersection of the lines was taken as the mark, and the 
section in which that appeared was the score assigned. Finally, for scales in 
which the participant had not marked the subscale, the mean of the other scores 
was substituted. For each factor, the weight was multiplied by the score on the 
subscale to get the factor score. Finally, the factor scores were added together 
and that sum was divided by the number of weights, which were usually 15. This 
number was the total workload score. 

C. PARTICIPANTS 

The participants were randomly assigned from the twenty students of the 
NPS who volunteered for the study. All of the participants were male. Nine 
participants had a strong aviation background (1 helicopter pilot, 8 fighter-jet 
pilots). The mean age of the pilots group was 31 (range: 27 to 40) and the 
average flight experience was flight hours (range: 60 to 2500). The mean age of 
the group of non-pilots was 35 years (range: 27 to 39). 
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V. DATA ANALYSIS 


A. HYPOTHESES 

1. Primary Hypotheses 

The primary hypothesis states that an operator will be able to assess 
spatial position significantly faster and with grater accuracy using the proposed 
instruments (Weber Box) compared to use of the traditional instruments when 
measured under increased workload conditions. This hypothesis can be explored 
via the objective results of the static spatial awareness experiment. The 
measured time it takes a participant to assess a particular spatial situation 
indicates the participant’s ability to create a mental model of the current spatial 
situation. The measured angular errors (for pitch and roll) from the given spatial 
orientation indicated the accuracy of this mental model (Weber, 2006). 

The first hypothesis is that there will be a significant difference in the time 
required to assess the situation, that is, delta time= time (proposed design) - 
time (traditional design). The second hypothesis states that participants this time 
difference also will be true in the increased workload condition. We derive the 
null-hypothesis that there is no proportional difference between the orientation 
times of the two designs under an increased workload. Hence, the null- 
hypothesis H 0 is: Hd = [fh _ a] ar| d M D ~ 0, and therefore (3i= 0, that is, the slope 

of the linear regression model is zero. The alternative hypothesis is that the 
orientation time using the proposed design is proportionally shorter than using 
the traditional design under increased workload. Hence, H a is:// D <0, and 

therefore Pi < 0, that is, the slope of the linear regression model is negative. 

2. Secondary Hypothesis 

As mentioned before, the measured angular errors (for pitch and roll) from 
the given and therefore true spatial orientation indicated the accuracy of this 
mental model. Thus, our secondary hypothesis was that the angular precision of 
the assessed spatial situation was at least at the same level for both instrument 
designs under increased workload. 
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A negative sum of angular errors indicated a higher accuracy in the spatial 
awareness assessment. Equal accuracy indicates that the new design was at 
least as efficient as the traditional one, even while operating under an increased 
workload. The precision was measured in terms of angular errors in 15° steps. 
This means that errors within 15° were rounded to the next value. For instance 
an error angle of 17° would be counted as a 15° error and a 38° error would be 
counted as a 45° error and so on. 

To determine the number of participants needed in the study, we used 
statistical software package JMP® version 5.1.2. In this Alpha is the significance 
level, in this study 0.05, and this implies willingness to accept (if the true 
difference between groups is zero) that 5% (alpha) of the time a significant 
difference will be incorrectly declared. Error Std Deviation is the true residual 
error and Difference to detect is the smallest detectable difference, (how small a 
difference you want to be able to declare statistically significant) in this study 1.4 
sec. Sample Size is the total number of observations (runs, experimental units, or 
samples) and Power is the probability of getting a statistic that will be declared 
statistically significant in this study 0.85. The estimate of the number of 
participants was based on the results of the pilot study. 
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Figure 12. Statistics of required sample number 


B. GENERAL DATA ANALYSIS 

1. General Data Description 

The data analysis was supported by statistical software package JMP® 
version 5.1.2 . The categories were: 

• Orientation Time: the time a participant needs to build his/her 
spatial awareness in seconds. 

• Pitch Error: the angular error of the estimated pitch in degrees 

• Roll Error: the angular error of the estimated roll of the aircraft in 

degrees. 

• Demographic questionnaire: age in years and flight experience in 
hours. 

• Post-Experimental Questionnaire: the subjective assessment of the 

experiment. 

2. Demographic Data 

The non-pilot participants and the participant with aviation background 

have similar means of the age. The detailed statistics are shown in Figure 14. 

41 









































Mean 
Std Dev 
Std Err Mean 
upper 95% Mean 
lower 95% Mean 
N 

Sum Wgt 

Sum 

Variance 

Skewness 

Kudos is 

CV 

N Missing 


32.5 

4.5022003 

0.2005212 

33.19401 

32.40599 

480 

480 

15744 

19.300209 

-0.214051 

-0.530567 

13.393903 

0 


Figure 13. Statistics of participant’s age 
The overall statistics for the age are as follows: 



Mean 

StDev 

Minimum 

Maximum 

Age 

32.5 

4.502 

27 

40 

Pilots 

35 

3.887 

27 

39 

Non-Pilots 

31 

3.558 

27 

40 


Based on the statistics of the flight hours, we were able to conclude that 
the group of pilots can be considered as experienced. Most of the pilot 
participants had more than 1000 flight hours; only three of the nine had less than 
1000 flight hours of experience. The detailed statistics of flight experience are 
shown in Figure 15. 
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XX 



Mean 

595.5 

Std Dev 

902.98586 

Std Err Mean 

41.215477 

upper 95% Mean 

676.48548 

lower 95% Mean 

514.51452 

N 

480 

Sum Wgt 

480 

Sum 

285840 

Variance 

815383.47 

Skewness 

1.1688686 

Kurtosis 

-0.285463 

CV 

151.63491 

N Missing 

0 


Figure 14. Statistics of participant’s flight experience 


3. Orientation Time Effects 

The comparison of the means and the distribution graphs does not reveal 
information about individual differences between the participants but based on a 
Tukey - Kramer comparing means model it is revealed that the order which the 
participants started the experiment, meaning starting with the set A of the 
traditional instrument or set B of Weber Box, basically made no difference in 
orientation time as shown in Figure 16. 
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Figure 15. Linear regression on Orientation time (all participants) 


Also we can state that there were no differences between the group of 
pilots and non-pilots. That is, pilots were slightly faster than non-pilots were (0.45 
seconds). Besides the practical irrelevance of this difference, since being a pilot 
was a non-significant factor in the overall linear regression model, this difference 
has no statistical relevance as shown in Figure 16. 
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Pilot? (0/1) 0.05 


Figure 16. Linear regression on Orientation time (all participants) 

When we compare orientation time for the two designs, it is clearly 
observable that the new design achieves much shorter orientation time. In 
addition we were able to determine that the variance of the data decreased by 
using the proposed design. 

The basic evaluation by new design by orientation time, with a level of 
significance of a = 0.01 by providing a test statistic with F (1,478) = 703.1474, p < 
0.0001 achieved an adjusted R 2 of 0.5953, which states that this model explains 
60% of the variances in the data. This was an acceptable value to establish 
statistical significance. Figure 18 shows the distance between the two sets of 
orientation time for the linear regression model. The difference in the means of 
the original data was negative 7.95 seconds. Thus, it took an operator on 
average the 3 times longer to interpret the traditional instruments. 
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Figure 17. Linear regression on Orientation time (all participants) 


The data revealed a significant difference in test times between the two 
set ups (operating one or two UAVs simultaneously). The basic evaluation of 
workload used by orientation time, with a level of significance of a = 0.01 by 
providing a test statistic with F (1,478) = 150.8302, p < 0.0001 we achieved an 
adjusted R 2 of 0.2398, which states that this model explains over 20% of the 
variances in the data. Figure 18 shows the distance between the two sets of 
orientation time for the linear regression model. 

The difference in the means of the original data was 5.05 seconds. Thus, 
it took an operator, on average, 2 times longer to interpret the flight instruments 
operating under increased workload. 
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Tukey-Kramer 

Number of UAVs (0/1) 0.05 

Figure 18. Linear regression on Orientation time (all participants) 

We were concerned that there would be a learning effect present because 
participants executed 6 tasks per set up, and it is reasonable to expect that they 
might acquire proficiency as they completed more tasks. Figure 19 presents an 
ANOVA for task time using task number (1-6) as an independent variable. Using 
a t-test, there was no significant difference between times for tasks 1 through 6. 
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Figure 19. No learning effect as participants progressed through the tasks for 

(all participants) 


4. General Statistical Methodology 

To investigate the influences of the designs on orientation time, we 
decided to conduct a multiple regression analysis. First, we conducted a 
stepwise regression analysis (full factorial) to eliminate factors that were not 
influential in our model. The regression model that we are developed associates 
the average response to the decision factors. We began by using only main 
effects to get an idea of the impact of these factors alone on our model. We 
developed the main-effects model using the mixed stepwise function in JMP. 
This function uses alternating forward and backward steps, allowing terms to 
enter the model on the forward step and leave the model on the backward step, 
based on a significance level for each. For our model we allowed terms to enter 
the model at a significance of .25 and removed terms with significance less than 
.01. The potential predictors were workload and instrument design as the main 
suspected factors, and aviation experience (pilot or not), order of presentation 
and participant’s age. The participants were handled as a blocking factor for the 
regression model. The response variables were orientation time, and pitch and 
roll error time. For the response variable orientation time only the display design 

and the workload had significant influence as a predictor variable for all response 
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variables with F (5,475) = 1303.975, p < 0.000 and F (5,475) = 652.783, p < 
0.000 respectfully. However, the graph showed that the display design 
contributed the most with an R 2 =0.83 while the workload explained the data with 
an R 2 =0.25. We conducted stepwise regression for the rest of the response 
variables with similar results. 
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Figure 20. Stepwise regression analysis for orientation time 


The regression model enables us to reject the Ho at a level of significance 
of a = 0.01 by providing a test statistic with F(15,464) = 726.8835 (sufficiently 
large), p < 0.0001, with R 2 of 0.9591. 
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RSquare 0.959181 

RSquare Adj 0.957861 

Root Mean Square Error 1.060053 

Mean of Response 7.282292 

Observations (or Sum Wgts) 480 
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DF 

Sum of Squares 
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15 
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816.808 

726.8835 

Error 

464 

521.402 

1.124 

Prob > F 

C. Total 

479 

12773.519 


<.0001 


Figure 21. Graph explaining test time as a result of the independent variables 


A more extensive analysis allowed us to conduct an interpretation with the 
Prediction profiler (displays prediction traces for each X variable. A prediction 
trace is the predicted response as one variable is changed while the others are 
held constant at the current values) in JMP where an interaction plot is the 
evidence of interaction, showed by nonparallel lines. The importance of a factor 
can be assessed to some extent by the steepness of the prediction trace. If there 
are interaction effects or cross-product effects in the model, the prediction traces 
can shift their slope and curvature as you change current values of other terms. If 
there are no interaction effects, the traces only change in height, not slope or 
shape. We can notice through all the different combinations, that among the 
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following idependent variables only the instrument design (new design (0/1)) and 
workload (number of UAVs(0/1)) do shift the slope and change the values of the 
terms, with the first having the most influence. 
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Figure 22. Prediction profiler analysis for orientation time 

5. Angular Pitch and Roll Errors Effects 

The statistics of the pitch and roll errors show that participants were able 
to judge the current pitch angle of the UAV more accurately using the new design 
and as expected they performed even better while operating under an increased 
workload as it is shown in the Figure 23 and 24 (following). The box-plot of the 
data suggests a significant reduction of the pitch errors by the new design which 
is even more evident while operating under an increased workload. 

Although the data revealed that there was no significant difference in 
errors between the two set ups (operating one or two UAVs simultaneously). The 
difference in the means of the original data for pitch and roll by design (0/1) was 
18.25° and 01.37° and 06.25° and 0.7 respectively. Thus, an operator on average 
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made the same proportional errors operating the new design under an increased 
workload while he increased the errors operating with the traditional instruments 
set up. 



0 120 18.5000 1.9711 

1 120 36.7500 1.9711 



New design 

Level Number Mean Std Error 

0 120 2.25000 0.82927 

1 120 3.62500 0.82927 


Figure 23. Linear regression for pitch time by design (0/1) 




Traditional design New design 


Level 

Number 

Mean 

Std Error 

Level 

Number 

Mean 

Std Error 

0 

120 

76.3667 

4.0714 

0 

120 

6.37500 

1.5506 

1 

120 

84.1250 

4.0714 

1 

120 

7.00000 

1.5506 


Figure 24. Linear regression for roll time by design (0/1) 
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We can notice through all the different combinations of independent 
variables, only the instrument design (new design (0/1)) and workload (number of 
UAVs(0/1)) do shift the slope and change the values of the terms, with instrument 
design having the mostinfluence. 
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Figure 25. Prediction profiler analysis for pitch and roll time 
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The distribution of the angular pitch and roll error can be assumed as 
normal. However, the box-plots show some outliers, which we decided to keep in 
the dataset. These outliers represent, in our opinion, human errors, which may 
occur and hence were not an abnormality. In particular angular error can lead in 
many cases to serious aviation mishaps (Previc et al., 2004). 

The error in judging their own roll angle was mastered by pilots and non¬ 
pilots with an equal level of accuracy and a similar variance of the data. Pilots 
were slightly better in estimating the pitch and roll of the UAV. 

6. Post-Experiment Questionnaires 

The analysis of the NASA-TLX questionnaire (of the subjective workload 
of all the operators as a group) showed a significantly higher workload in the 
increase workload setup. When analyzed by the standard of aviation experience, 
results indicated that those with aviation experience reported significant 
differences in workload between setups, while those without aviation experience 
reported more significant differences. This indicates that the operators’ subjective 
workload of operators’ increases with the addition of secondary tasks, but this 
may vary with aviation experience. The mean NASA-TLX scores are presented in 
Table 1. 


Table 1. Mean Subjective Workload Scores 



1 UAV TLX score 

2 UAVs TLX score 

All participants 

62.88 

65.48 

Pilots 

58.24 

61.38 

Non pilots 

64.82 

66.53 


The difficulty of the tasks was experienced as slightly difficult to somewhat 
easy to accomplish. The answers of the post-experimental questionnaires 
showed a strong tendency towards the preference of the Weber Box. However, 
the questionnaires show that most participants (over 80%), pilots and non-pilots, 
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appreciated the proposed design and felt supported in their spatial orientation 
tasks more than using the traditional setup (basic T). The answers in 
percentages were as follows. 


Table 2. Results of Post-Experiment Questionnaires - Orientation Task A 


Question 

Answer-Options 

In general, how do you judge the 
difficulty of evaluating the current 
attitude of the UAV? 

Very 

easy 

5% 

Somewhat 

easy 

40%) 

Border 

line 

20% 

Somewhat 

complicated 

35% 

Very difficult 

0% 

In general, how do you judge 
difficulty of using the traditional flight 
instruments for this task? 

Very 

easy 

0% 

Somewhat 

easy 

20% 

Border 

line 

20% 

Somewhat 

complicated 

50% 

Very difficult 

15% 

In general, how do you judge the 
difficulty of using the WEBER-Box for 
this task? 

Very 

easy 

50% 

Somewhat 

easy 

50% 

Border 

line 

0% 

Somewhat 

complicated 

0% 

Very difficult 

0% 

Did you feel better aware of the 
spatial orientation of the UAV with 
the WEBER-Box in comparison with 
the traditional flight instruments? 

Much less 

0% 

Somewhat 

less 

0% 

Border 

line 

0% 

Somewhat 

more 

25% 

Much more 

75% 

Did having the WEBER-Box increase 
or decrease your monitoring 
demands in comparison with the 
traditional flight instruments? 

Greatly 

decreased 

45% 

Somewhat 

decreased 

50% 

Unaffected 

5% 

Somewhat 

increased 

0% 

Greatly 

increased 

0% 

Did having the WEBER-Box increase 
or decrease your overall workload in 
comparison with the traditional flight 
instruments? 

Greatly 

decreased 

45% 

Somewhat 

decreased 

50% 

Unaffected 

5% 

Somewhat 

increased 

0% 

Greatly 

increased 

0% 

Did having the WEBER-Box increase 
or decrease your overall frustration 
level (irritate, stress, insecure) in 
comparison with the traditional flight 
instruments? 

Greatly 

decreased 

70% 

Somewhat 

decreased 

25% 

Unaffected 

5% 

Somewhat 

increased 

0% 

Greatly 

increased 

0% 

How would you characterize your 
Computer Skills 

None 

0% 

Fair 

0% 

Average 

10% 

Good 

70% 

Excellent 

20% 

How much experience do you have 
with computer games? 

None 

Fair 

Average 

30% 

Good 

60% 

Excellent 

10% 
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VI. CONCLUSIONS 


A. OVERALL ASSESSMENT OF THE STATISTICAL RESULTS 

The purpose of the current study was to assess both subjective workload 
and associated performance decrements while operating the proposed display 
design versus the traditional instrument design (Weber, 2006). Specifically, the 
focus was to assess the relationship between a subjective workload and various 
objective measures of performance both with and without the introduction of a 
secondary task involving operating one or two UAVs. 

The statistical analysis supported the proposed design in all aspects. The 
design seemed to sustain remarkably well the spatial awareness in 3D 
orientation tasks even under high workload conditions. Time to assess a spatial 
situation decreases significantly, with the new display, whereas accuracy of this 
spatial judgment at least maintains its level. Judgment errors were minimized and 
the extreme errors were almost eliminated. Figure 26 shows the level of 
increased orientation time as workload increases while operating the traditional 
design. The steeper the line shows, the more influence is of the added workload. 
It seems that the pilots were not able to improve their speed better than non¬ 
pilots but further analysis revealed that was due to increased accuracy. 




Figure 26. Linear regression for orientation time by design (traditional) 
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Figure 27 shows the level of increased orientation time as the workload 
increases while operating the proposed design. The added workload is more 
influential when the line is steeper. It seems that the pilots were able to improve 
their speed better than non-pilots did. 



Figure 27. Linear regression for orientation time by design (proposed) 


It was surprising that pilots and non-pilots were so similar. The differences 
were minor. We explain this phenomenon with the character of the tests, which 
were not overly demanding. Another aspect might be that the underlying 
cognitive processes are basically the same whether operating the Weber Box 
design (while it was almost twice as big) or operating the setup with the 
traditional instruments. Thus, everybody took equal advantage out of the 
proposed design. 


The statistics of the values were as follows: 


Traditional design 

Source 

DF 

F Ratio 

Prob > F 


Number of UAVs 1 

1655.561 

<.0001 


(0/1) 





Error 

238 




C. Total 

239 




Level Number 

Mean 

Std Error 

Lower 95% 

Upper 95% 

0 120 

7.0550 

0.14624 

6.767 

7.343 

1 120 

15.4700 

0.14624 

15.182 

15.758 
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Proposed design 

Source 

DF 

F Ratio 

Prob > F 


Number of UAVs (0/1) 

1 

295.2498 <.0001 


Error 

238 




C. Total 

Level Number Mean 

239 

Std Error 

Lower 95% 

Upper 95% 

0 120 2.45667 

0.06958 

2.3196 

2.5937 

1 120 4.14750 

0.06958 

4.0104 

4.2846 


The proposed study was conducted to support operators of UAVs in 
extreme situations - increased workload. The main goal was to provide an 
intuitive way to improve the level of spatial awareness in ambiguous conditions. 
In particular, it utilizes the principle of the mental model by introducing an 
exocentric view on a virtual avatar in an abstract three-dimensional virtual world 
while the participants were subjected to increased workload. This positive picture 
was supported by the comments and post-experimental questionnaires. The 
majority of the participants found the “WEBER-Box” to be very helpful and 
intuitive to understand, especially under increased workload where the response 
time was proportionally smaller. The participants felt that the “WEBER-Box” 
improves their spatial awareness and made it easier to accomplish their tasks. 
The following Figure 28 depicts JMP’s partition platform which enables us to 
systematically analyze data sets and to discover unsuspected or unknown 
relationships by visualizing a successive tree of partitions according to a 
relationship between the X and Y variables. It finds a set of cuts or groupings of 
X values that best predict a Y value by exhaustively searching all possible cuts or 
groupings, recursively. Thus we can easily see, achieving R 2 of 0.959, the impact 
of the design as opposed to the increased workload. Similar partition model were 
found for roll and pitch errors indicating the superiority of the Weber box. 
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Figure 28. Partition for orientation time 


Many of the underlying interactions and effects are not completely 
investigated. Questions about design improvements and practical applications 
need to be answered as well. However, by applying human-centered design 
principles we were able to design an efficient tool to support spatial awareness in 
3D-orientataion tasks. Our data clearly reveals that the nontraditional design 
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Weber box (Weber, 2006) yielded superior performance to the traditional one. 
The design was accepted and appreciated by both the pilot and the non-pilot 
participants and was proven beneficial especially under complex tasks with 
increased workload. 

B. FUTURE APPLICATIONS 

We can see applications of the WEBER Box design in all major fields of 
aviation. Since this design is intuitively understandable, it might help flight 
students to understand traditional flight instruments. Thus, integration into 
traditional flight instruments or even to add flight instruments to WEBER Box 
might be beneficial. Because this study’s major goal was to implement, test and 
evaluate a novel display design (Weber, 2006), it turned out that more questions 
were formed than it answered. Foremost questions that are open to future 
research are about design improvements and the influence of alternations and 
simplifications. Questions may include: 

• What is the optimal size? 

• How much can we simplify the aircraft model? 

• Should we integrate scales to make it a complete flight instrument 
and given that, how much information and graphics can be 
simplified without degrading the outcome? 

• Are the principles of the WEBER Box applicable to other fields (car 
driving, under water robots, etc.)? 

• Is the WEBER Box useful for training purposes as pilot training, 
UAV ground control systems operators, etc.? 

• How much an even more increased workload environment (i.e. 
auditory tasks) will affect performance and spatial attitude 
definition? 

Other research might investigate the potential of the current design, the 
regarding the altitude representation and the various possible implementations of 
warning at extreme situations. 
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APPENDIX A. IRB APPLICATION AND APPROVAL 



Anthony Ciavarelli, Ed.D. 

Modeling Virtual Environment and Simulation 
Department 
Watkins Hall 

Naval Postgraduate School 
Monterey, California 93943 


831-656-2191 
DSN: 756-2191 
Fax: 831-656-2595 
ACiavarelli@nps.edu 


To: Protection of Human Subjects Committee 

Subject: Application for Human Subjects Review for Use of Avatars to 
Support and Enhance Spatial Awareness in 3D-Orientation-Tasks 


1. Attached is a set of documents outlining a proposed experiment to be 
conducted over the year to support the thesis of Lieutenant Colonel Dimitrios 
Myttas, GR. 

2. We are requesting approval of the described experimental protocol. An 
experimental outline is included for your reference that describes the methods 
and measures we plan to use. 

3. We include the consent forms, privacy act statements, questionnaires, and 
briefing forms we will be using in the experiment. 

4. We understand that any modifications to the protocol or 
instruments/measures will require submission of updated IRB paperwork and 
possible re-review. Similarly, we understand that any untoward event or injury 
that involves a research participant will be reported immediately to the IRB 
Chair and NPS Dean of Research. 


Anthony Ciavarelli 
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APPLICATION FOR 
HUMAN SUBJECTS REVIEW 


HSR NUMBER 


(HSR) 


(to be assigned) 


PRINCIPAL INVESTIGATOR(S) (Full Name, Code, Telephone) 

Anthony Ciavarelli, Ed.D., Monterey, CA 93943, Phone: 831 656-2191 
(Thesis LtCol Dimitrios Myttas, GR) 


APPROVAL REQUESTED [ X ] New [ ] Renewal 


LEVEL OF RISK [ ] Exempt [ ] Minimal [ ] More than Minimal 
Justification: 3D-Virtual Environment (flight simulator software) 


WORK WILL BE DONE IN (Site/Bldg/Rm) 

NPS, Watkins Hall, Rm 212B 

ESTIMATED NUMBER OF DAYS TO 
COMPLETE 

10 

MAXIMUM NUMBER OF SUBJECTS 

30 

ESTIMATED LENGTH OF EACH 
SUBJECT’S PARTICIPATION 

2 hours 

SPECIAL POPULATIONS THAT WILL BE USED AS SUBJECTS 


[] Subordinates [] Minors [ X ] NPS Students [] Special Needs (e.g. Pregnant 
women) 

Specify safeguards to avoid undue influence and protect subject’s rights: none 


OUTSIDE COOPERATING INVESTIGATORS AND AGENCIES 


- none- 

[ ] A copy of the cooperating institution’s HSR decision is attached. _ 

TITLE OF EXPERIMENT AND DESCRIPTION OF RESEARCH (attach additional 
sheet if needed). Methodology attached 


I have read and understand NPS Notice on the Protection of Human Subjects. If there 
are any changes in any of the above information or any changes to the attached 
Protocol, Consent Form, or Debriefing Statement, I will suspend the experiment until I 
obtain new Committee approval. 

SIGNATURE_ 

DATE 
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APPENDIX B. EXPERIMENT PROTOCOL 


EXPERIMENTAL WORKSTATION ENVIRONMENT STARTUP 


1 . 

2 . 

3. 

4. 

5. 


6 . 


Switch on the personal computer (laptop), and the extra 17” TFT monitor 
on the experimenter desk. 

Login at the computer by pressing ALT_DEL_CTRL. Let the password 
empty and press ENTER. 

Open the “Local Area Network” icon in the task bar using the RIGHT 
mouse button. 

In the following menu chose STATUS, click on the SUPPORT-tab and 
write down the IP-address (e.g. 131.120.151.6), or just type “ipconfig” at 
command prompt. 

Start on the desktop the following programs by clicking the icons twice by 
the mouse: 

a. “UDP Reflector” => type in the 
dialog box the IP-address of the 
computer und press ENTER 

b. “X-Plane Experiment” 

After “X-Plane” started, 


m 


IP Address of 2nd Computer: 
131.120.XXX.XXX 


ll 27.0.0.1 



choose menu “Location” => select “Place 
Aircraft by Airport” => chose “Twenty-nine 
Palms EAF” => click “Go to this Airport” 
choose menu “Files” => select “Load 
Scenarion” => click on “Experiment.sit” 
wait until the situation is loaded, than hit the 
“P” button of the keyboard to pause the 
simulation 

make sure that the following values have 
been typed (127.0.0.1 - localhost). 


Start the WEBER BOX 

1. On the computer double click on the icon "Start Weber Box” icon -> in 
the window type in for the UDP port: 49002 and 1.0 for the speed. Now the 
WEBER-Box should appear. The experimenter can drag it to the 17” monitor 
for all experiments which demand the WEBER-Box. 

2. Double-click on the icon “Experimenter” to start the input consol. 

=> Type in the field “IP-Address” the number 

131.120.150.248 and click the button “set” 
right to the input field 

=> Type in the filed “ID” the number 16 and 
click the button “int” below the field 
=> Insert the different values (V0, VI- Pitch, 
Roll) and click the button “fit” below the field 


KPI«mPAi»»mi| , 0001 




♦ ♦ ♦ ♦ ♦ ♦ ♦ ♦ 
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PRE-EXPERIMENT PROTOCOL 

Check if all forms are prepared and signed by the experimenter and the 

principle investigator 

Escort participant to peroration desk 

Administer Initial Questionnaires 

o Hand out the written experiment objectives and the experiment 
introduction. 

o Present Institutional “IRB Participant Consent Form”, the “Minimal 
Risk Consent Statement”, the “Privacy Act Statement” and let it 
sign by the participant. 

o Check if the participant’s ID number is noted on every page of the 
initial questionnaire and present it and let the participant sign it. 
Request if the participant has any further questions 

SPATIAL AWARENESS EXPERIMENT PROTOCOL 

Participant is led to the Experimental workstation and shown all 
equipment. 

Participant is instructed to sit in front of the operator computer. 
Experimenter explains all flight instruments and controls incl. the Weber 
Box. 

Experimenter starts the training scenario (the aircraft is airborne on a safe 
altitude and attitude) and allows the participant to fly for about 10 minutes. 
In this time the experimenter provides any help regarding explanations of 
the use of the flight instruments and controls. He must not refer to any of 
the later experiment tasks. 

Experimenter gives the participant the briefing for the first task 
(Recognition of spatial orientation of the aircraft) and makes participant 
familiar with the procedures. He is now advised to read the task 
instructions for this task. 

Experimenter switches off the instrumentation monitor or switches off the 
Weber Box respectively, depending if he begins with or without using the 
Weber Box. 

Experimenter asks participant if he/she understands everything and if he 
may start the experiment 

The first trial is a test trial; the test-trial sheet has to be used for it. 
Experimenter: 

o starts spatial awareness scenario 
o prepares the stop watch 

Experimenter: Switches off the monitor, and determines a set of pitch- and 
roll-angles from the task sheet “Test A” (“Test B” for the 2 nd set of 
experiments) by 

o Typing the angle values in degrees in field “00” for the pitch and 
field “01” for the roll and clicks on the button “fit” below the field to 
store the values 



o Clicks the button “Send Data Package” to transfer the values to the 
experimental workstation monitor 

o Observes whether the flight simulation software (X-Plane or Weber 
Box) reacts as expected 

o If not, he checks again all values and presses all described buttons 
again to store the data 

• Experimenter: 

o Stands directly on the participants left side and places his left hand 
at the power button of the monitor and holds the stop watch in his 
right hand 

o Switches on the monitor and tells the participant that he may start 
to look at the instruments 

o In the moment the participant starts looking the monitor and at the 
instruments, he starts the stop watch 
o When the participant says “Stop!” he switches of the monitor and 
stops the stop watch at the same time 

• Experimenter: 

o Let the participant draw his/her opinion about pitch and roll into the 
provided schema on the evaluation sheet 
o Writes the measured time into field next to the evaluation schema, 
o Experimenter starts over with the next experiment loop and a new 
set of pitch and roll angles until the participant finished six tasks 
(plus one test trial). 

• Experimenter starts with the SPATIAL AWARENESS EXPERIMENT with 
increased workload (Operating two UAVs) 

POST EXPERIMENT PROTOCOL 

• Participant is allowed to have a five minutes break 

• Reseat the participant and start with the 2 nd block of experiments including 
the spatial awareness experiment and spatial awareness experiment with 
increased workload. 

• After all tests are completed the experimenter: 

o Thanks the participant for his/her effort 
o Guides participant to a chair 

o Provides the “Post-Experiment Questionnaire” and asks participant 
to fill it out 

o Evaluates the angular errors of the SA-experiment and notes it 
down into the evaluation sheets 
o Records all results in the participant’s database. 

• Thank participant for participating in the experiment. 
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APPENDIX C. PARTICIPANT INFORMATION FORMS 


PARTICIPANT CONSENT FORM 

1. Introduction. You are invited to participate in a study of spatial awareness of virtual 
environments. With information gathered from you and other participants, we hope to 
prove the concept of the use of virtual avatars to support and enhance spatial 
awareness and improve orientation performance. We ask you to read and sign this 
form indicating that you agree to be in the study. Please ask any questions you may 
have before signing. 

2. Background Information. The Naval Postgraduate School MOVES Institute is 
conducting this study. 

3. Procedures. If you agree to participate in this study, the researcher will explain the 
tasks in detail. There will be two sessions with two different orientation tasks each. 
The execution phases will last approximately two hours total, during which you will be 
asked to accomplish a number of tasks related to spatial awareness while operating a 
virtual flight simulator software. Following the study you will be asked to fill out 
questionnaires during a 20-minutes rest period. 

4. Risks and Benefits. This research involves having a participant fly a 3D virtual 
flight simulator software. If you have any cardiac risk factors (High blood pressure, 
smoking, diabetes, high cholesterol, previous heart problems), we request that you 
PLEASE INFORM THE EXPERIMENT ADMINISTRATOR AND YOU WILL NOT BE 
ABLE PARTICIPATE IN THE EXPERIMENT. 

5. Compensation. No tangible reward will be given. A copy of the results will be 
available to you at the conclusion of the experiment period. 

6. Confidentiality. The records of this study will be kept confidential. No information will 
be publicly accessible which could identify you as a participant. 

7. Voluntary Nature of the Study. If you agree to participate, you are free to withdraw 
from the study at any time without prejudice. You will be provided a copy of this form 
for your records. 

8. Points of Contact. If you have any further questions or comments after the 
completion of the study, you may contact the research supervisor, Dr. Anthony 
Ciavarelli, Ed.D., NPS Monterey, CA 93943, Phone: 831 656-1073 

9. Statement of Consent. I have read the above information. I have asked all questions 
and have had my questions answered. I agree to participate in this study. 


Participant’s Signature 


Date 


Researcher’s Signature 


Date 
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MINIMAL RISK CONSENT STATEMENT 

NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 

Participant: VOLUNTARY CONSENT TO BE A RESEARCH PARTICIPANT IN: 

Use of Avatars to Support and Enhance Spatial Awareness in 3D- 

Orientation-Tasks 


1. I have read, understand and been provided "Information for Participants" that 
provides the details of the acknowledgments below. 

2. I understand that this project involves research. An explanation of the purposes of 
the research, a description of procedures to be used, identification of experimental 
procedures, and the extended duration of my participation have been provided to 
me. 

3. I understand that this project does not involve more than minimal risk. I have been 
informed of any reasonably foreseeable risks or discomforts to me. 

4. I have been informed of any benefits to me or to others that may reasonably be 
expected from the research. 

5. I have signed a statement describing the extent to which confidentiality of records 
identifying me will be maintained. 

6. I have been informed of any compensation and/or medical treatments available if 
injury occurs and is so, what they consist of, or where further information may be 
obtained. 

7. I understand that my participation in this project is voluntary: refusal to participate will 
involve no penalty or loss of benefits to which I am otherwise entitled. I also 
understand that I may discontinue participation at any time without penalty or loss of 
benefits to which I am otherwise entitled. 

8. I understand that the individual to contact should I need answers to pertinent 

questions about the research is Professor Anthony Ciavarelli, Thesis Advisor, and 
about my rights as a research participant or concerning a research related injury is 
Prof. Rudy Darken, MOVES Institute Chairman or the NPS IRB Medical Advisor, 
LTC Eric Morgan, MC, USA, Presidio of Monterey, (831) 242-7550, 

eric.morgan@NW.AMEDD.ARMY.MIL 


Signature of Principal Investigator Date 


Signature of Volunteer 


Date 
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PRIVACY ACT STATEMENT 


NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 
PRIVACY ACT STATEMENT 

1. Purpose: Spatial cognition data will be collected to enhance knowledge, and 
to develop tests, procedures, and equipment to improve the development of 
Virtual Environments. 

2. Use: Spatial cognition data will be used for statistical analysis by the 
Departments of the Navy and Defense, and other U.S. Government 
agencies, provided this use is compatible with the purpose for which the 
information was collected. Use of the information may be granted to 
legitimate non-government agencies or individuals by the Naval 
Postgraduate School in accordance with the provisions of the Freedom of 
Information Act. 

3. Disclosure/Confidentiality: 

a. I have been assured that my privacy will be safeguarded. I will be 
assigned a control or code number, which thereafter will be the only 
identifying entry on any of the research records. The Researcher will 
maintain the cross-reference between name and control number. It will be 
decoded only when beneficial to me or if some circumstances, which is not 
apparent at this time, would make it clear that decoding would enhance the 
value of the research data. In all cases, the provisions of the Privacy Act 
Statement will be honored. 

b. I understand that a record of the information contained in this 
Consent Statement or derived from the experiment described herein will be 
retained permanently at the Naval Postgraduate School or by higher 
authority. I voluntarily agree to its disclosure to agencies or individuals 
indicated in paragraph 3 and I have been informed that failure to agree to 
such disclosure may negate the purpose for which the experiment was 
conducted. 

c. I also understand that disclosure of the requested information, 
including my Social Security Number, is voluntary. 


Name, Grade/Rank (if applicable) 
[Please print] 


Signature of Volunteer 


Date 
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APPENDIX D. TASK DESCRIPTIONS 


Spatial Orientation 


Background 

Unmanned air vehicles are nowadays seen as an area of great importance in the 
aerospace industry. An essential part of these aeronautical systems is the 
ground control station or GCS. This is the unit on the ground that sends and 
receives signals from one or several airborne units. These are normally very 
complicated systems that require many personnel. There is a general accepted 
way of thinking that says there are six main instruments, which give an overall 
picture of the aircrafts flight condition. These six primary instruments are 
airspeed indicator, altimeter, artificial horizon, direction indicator, vertical speed 
indicator and turn and bank indicator. It is important to group these instruments in 
the right way in order to let the pilot clearly interpret the situation without 
confusion. Over the years many different layouts have been experimented; 
however there is now one accepted layout, which all modern airplanes adopt. 
This is known as the basic T. 



Artificial Horizon 

This indicates the pitch, bank and heading attitude of the aircraft. Pitch, bank and 
heading attitude are represented by one moving element. This is a surface that 
symbolizes the natural horizon. This moves in three axes to indicate the change 
in all three parameters simultaneously. A fixed horizontal line on the indicator 
represents the aircraft. 

You are the operator of a combined rotor-/fixed-wing based Unmanned Aerial 
Vehicle (UAV) of type Eagle Eye. 
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The Eagle Eye has a 
wingspan of 15.2 ft, is 
17.9 ft in length, is 5.7 ft 
high, and weighs around 
2,000 pounds (depending 
on payload). 

The secure altitude is 
defined by 2500 feet. 
Below this altitude the risk 
of crashes and of being 
shot down by enemy fire 
soars. 


Experiment Goals 

The goal of this experiment is to test the two different flight instrument (traditional 
- Weber box) setups regarding their capabilities to support the operator of 
ground control station in controlling the aircraft. Of particular interest is to 
determine operator of increased workload and perception of unusual attitudes. 
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Experiment Schedule 

You will conduct two sets of trials. 

The order in which you will experience the two sets randomly assigned. The only 
difference between the sets is which set of flight instruments is used and the 
increased workload while operating two UAVs simultaneously. 

After the task briefing, you will get the opportunity to become familiar with 
operating the flight simulator software before you start with your assigned block 
of trials. After finishing your first block, you will have a short break to fill out a 
brief questionnaire, and then you will start with the second block of trials. 

After you have finished both sets of trials you will be de-briefed and asked to 
respond to a post-experiment questionnaire. 

Remember: 

• You are a volunteer - we truly appreciate your time and willing to 
participate at our experiment! 

• Please complete all trials if possible, but you can quit the experiment at 
any time if you need to. 

• In case you might experience discomfort, please do not hesitate to inform 
the experimenter and please do not hesitate to abort the experiment if the 
level of discomfort is too high for you. 

• All personal data will be handled confidentially and anonymously. 
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Experiment Schedule Diagram 
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Task One 


The UAV is on a test flight and enters a sector of high crosswinds. To simulate 
the broken data-link, you will be exposed to an unknown attitude of the aircraft. 

Experiment Implementation 

You will operate the UAV for 15 minutes or so. Periodically, the experimenter will 
give you the order to judge the current spatial attitude of the UAV. The simulation 
will freeze at this moment. 



EXPERIMENTER 


You will then hear the command “Stop Flying.” The monitor will be switched off 
and immediately after switched on again, simulating data link interruption. You 
will begin to evaluate your current spatial attitude using your flight instruments. 
When you feel you know what the current spatial attitude is like, say loudly 
“STOP.” At this moment the flight instruments will be switched off. 

Now you have to express the observed spatial attitude of the UAV relative to the 
ground. You will do this by determining pitch and roll which you will draw on an 
orientation chart. 

The time to accomplish this task and the accuracy of your spatial judgment will 
be measured. 

You will have one test trial to become familiar with procedure. 

Do you have any questions? 
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Task Two 


The UAVs are on a test flight and enter a sector of bad weather. The data-link 
connection to the UAVs was interrupted because of the weather conditions 
(simulated by switching off the monitor). When the data-link connection is 
reestablished, your task is to judge the current spatial attitude of the UAVs. The 
simulation will freeze at this moment. 


Experiment Implementation 

You will “fly” the UAVs for 15 minutes or so. Periodically, the experimenter will 
give you the order to judge the current spatial attitude of the UAVs. The 
simulation will freeze at this moment. 



EXPERIMENTER 


You will then hear the command “Stop Flying.” The Monitor will be switched off 
after immediately switched on again, simulating data link interruption. You will 
begin to evaluate your current spatial attitude using your flight instruments. When 
you feel you know what the current spatial attitude is like, say loudly “STOP.” At 
this moment the flight instruments will be switched off. 

Now you have to express the observed spatial attitude of the UAVs relative to the 
ground. You will do this by determining pitch and roll which you will draw on an 
orientation chart. 

The time to accomplish this task and the accuracy of your spatial judgment will 
be measured. 

You will have one test trial to become familiar with procedure. 

Do you have any questions? 
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APPENDIX E. QUESTIONNAIRES 


Initial Questionnaire 

Date: 


The information is required for research and your name will not be used therefore please answer as fully and 
truthfully as possible. 


Participant ID 


First Name: 


Middle Initial: 


Last Name: 


Age: 


Gender: 


Nationality: 


Male 


□ 


Female □ 


Are you experienced in aviation in any 
regard (incl. playing Flight Simulation 
Games)? 

YES □ NO □ 

Are you an aviator/pilot? 

YES □ NO □ 

Are you familiar with the basic set of 
flight instruments? 

YES □ NO □ 

Are you familiar with the basic flight 
controls to steer an airplane/control 

UAV? 

YES □ NO □ 

Did you operate any of these aircraft?: 
Fixed-Wing (Jet/Propeller), 

Rotor-Wing Aircraft, UAV 

Fixed □ Rotor □ Jet □ 

Propeller □ UAV □ 

How many flight hours do you have? 

Hours 

How many hours in a flight simulator do 
you have? 

Hours 

How long ago was your last flight? 

Years Month(s) 

How long ago was your last use of a 
flight simulator? 

Years Month(s) 

Are you trained in procedures of 
recovery from unusual attitude? 

YES □ NO 

□ 
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Post-Experiment Questionnaire 


The information is required for research and your name will not be used therefore please answer as fully and 
truthfully as possible. 


1 Participant ID 


Question 

Answer-Options 

In general, how do you judge the 
difficulty of evaluating the current 
attitude of the UAV? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge 
difficulty of using the traditional flight 
instruments for this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge the 
difficulty of using the WEBER-Box for 
this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

Did you feel better aware of the 
spatial orientation of the UAV with 
the WEBER-Box in comparison with 
the traditional flight instruments? 

Much less 

Somewhat 

less 

Border 

line 

Somewhat 

more 

Much more 

Did having the WEBER-Box increase 
or decrease your monitoring 
demands in comparison with the 
traditional flight instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 

Did having the WEBER-Box increase 
or decrease your overall workload in 
comparison with the traditional flight 
instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 

Did having the WEBER-Box increase 
or decrease your overall frustration 
level (irritate, stress, insecure) in 
comparison with the traditional flight 
instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 

How would you characterize your 
Computer Skills 

None 

Fair 

Average 

Good 

Excellent 

How much experience do you have 
with computer games? 

None 

Fair 

Average 

Good 

Excellent 
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In General: 


Experiment Tasks: 


WEBER-Box: 
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APPENDIX F. EVALUATION FORMS 
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Participant ID: . 

WEBER BOX: YES □ NO □ 





Pitch 


Test... 


Roll 
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Participant ID: 


WEBER BOX: YES □ NO □ 



Test 
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Pitch and Roll Setups 


Test A 


Trial 

Pitch 

Roll 

D 

-65 

-150 

C 

-50 

50 

E 

-45 

100 

F 

45 

-65 

A 

-75 

120 

B 

-85 

-30 


Test B 


Trial 

Pitch 

Roll 

FandA 

45and-75 

-65and120 

EandC 

-45and-50 

100and50 

DandB 

-65and-85 

-150and-30 

AandC 

-75and-50 

120and50 

BandF 

-85and45 

-30and-65 

CandD 

-50and-65 

50and-150 
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APPENDIX G. NATIONAL AERONAUTICS AND SPACE 
ADMINISTRATION - TASK LOAD INDEX (NASA-TLX) 
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APPENDIX H. EXPERIMENT DATA 
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